Learning for stochastic dynamic programming

We present experimental results about learning function values (i.e. Bellman values) in stochastic dynamic programming (SDP). All results come from openDP (opendp.sourceforge.net), a freely available source code, and therefore can be reproduced. The goal is an independent comparison of learning methods in the framework of SDP. 1 What is stochastic dynamic programming (SDP) ? We here very roughly introduce stochastic dynamic programming. The interested reader is referred to [1] for more details. Consider a dynamical system that stochastically evolves in time depending upon your decisions. Assume that time is discrete and has finitely many time steps. Assume that the total cost of your decisions is the sum of instantaneous costs. Precisely: cost = c1 + c2 + · · ·+ cT , ci = c(i, xi, di), xi = f(xi−1, di−1, ωi), di−1 = strategy(xi−1, ωi) where xi is the state at time step i, the ωi are a random process, cost is to be minimized, and strategy is the decision function that has to be optimized. Stochastic dynamic programming is a control problem : the element to be optimized is a function. Stochastic dynamic programming is based on the following principle : Take the decision at time step t such that the sum ”cost at time step t due to your decision” plus ”expected cost from time steps t + 1 to T from the state resulting from your decision” is minimal. Bellman’s optimality principle states that this strategy is optimal. Unfortunately, it can only be applied if the expected cost from time steps t+1 to T can be guessed, depending on the current state of the system and the decision. Bellman’s optimality principle reduces the control problem to the computation of this function. If xt can be computed from xt−1 and dt−1 (i.e., if f is known) then this is reduced to the computation of V (t, xt) = Ec(t, xt, dt) + c(t+ 1, xt+1, dt+1) + · · ·+ c(T, xT , dT ) Note that this function depends on the strategy. We consider this expectation for any optimal strategy (even if many strategies are optimal, V is uniquely determined). Stochastic dynamic programming is the computation of V backwards in time, thanks to the following equation : V (t, xt) = infdt c(t, xt, dt) + V (t+ 1, xt+1), or equivalently V (t, xt) = inf dt [ c(t, xt, dt) + Eωt+1V (t+ 1, f(xt, dt, ωt+1)) ]